a 
DOCUMENT RESUME 


. v : 
‘ ‘gO: ¥ \ . 
ED 141 ou 5 i 3 EA 009 752 
AUTHOR Holléy, Freda M. , oe ae i, 
TITLE . Evaluation in the Seventies: what We Have Learned 
; About Program Development and Implementation. a 
INSTITUTION | Austin Independent School District, Tex. a eee: 
REPORT NO ' Pub*73-57 ar i ‘ : 
- PUB DATE » 76° ‘ ; / 
NOTE  43p. . “i / 
EDRS PRICE MNF-$0.83 HC-$2.06 Plus Postage. / . 
DESCRIPTORS | ptbeas opel Innovation; Bducat iogal Objectives; - 
Elementaly Secondary Education; , *Organizational 


‘Effectiveness; Personnel Needs;/ *Program Design; 
*Program’ Development; *Program ! Evaluation; Brogram 
Planning; School Systems; *Sta€f Utilization 


IDENTIFIERS *AuStin Independent saiita! a trict TX - 
ABSTRACT hag, ~ 

This panee.« ‘generated by thet wetaa Texas, 
‘Independent School District's Office of Research and Evaluation, 


offers a set of working hypotheses about what a school district must 
do in the implementation of programs to imptove the 
cost/effectiveness ratio of educational innbvatiogs. The author draws 
on three years of the Austin school district's experience (with 


evaluating program implementation and on recent Yiterature on program 


development and evaluation. Her analysis leads ‘her to concludé that 


school district programs will be improved by setting priorities ‘by 
which programs are approved Or disapproved, encouraging the school: 


_G@istrict. staff to understand how the district functions as ‘an. P 


organization and how the prognam implementjation process works, 

devising realistic program designs, and“making maximum use of 

evaluation information. (Author/DS) 
. F ‘ 


~~“ 


GEOR go i olobdiia i igi ii ii dol ia i gg iO ISG IOS iia iatolotoiaiog tak tec 


* ‘Documents acquired by ERIC,include many informal unpublished * 
* materials not available from other sour¢es. ERIC makes every effort * 
* to obtain the best copy available. Nevertheless, items of marginal * 
* reproducibility are. often encountered and this affects the quality 
* of the microfiche and hardcopy reproduétions'ERIC makes available 
*, via the ERIC Document Reproduction Service (EDRS): EDRS is not 

* responsible for ‘the quality of the original document. Reproductions 
* supplied by EDRS are the best’ that can be made from the original. 
ACERT LAAE LAA ERS ee ON RE ee Pn ep NS ee eee ae ee eee 


™ 


eH HH 


- ” a 
ny 
' : 


EDUCATION & WELFARE 
& NATIONAL INSTITUTE OF 


i! + 
‘ * . 
‘ oar) 
* i) . 
‘ P , P EDUCATION 

esed rc ee RARE ee eg pes etal 

: *, OUCED EKACTLY AS RECE VED FROM 
. i . ( rPANIZATIO 
’ 


THE PERSON OR ONE N ORIGIN 
; ATING T POINTS OF viEW OR OPINIONS 
, : F Fy TATED DO NOT NECESSARILY REPRE 
‘ ENT OFS AL NATIONAL NSTITUTE OF 
. t aii ? ED at NP * NOR RO Y 
‘ P . gsr . Py 
‘ 
' 
4 a “y 
’ i ‘ 
: : 2 : . 
E ’ Pa 
b a 
4! 3 ee ‘ ., 
. + . 
. 


‘ F : 
‘ 4 . 7 
= ‘ * +f. ¢ 
. 2 ae oe it bs ae 
‘ 5 ~sewel if ’ K ? a . aie whe Ns ; : te 
1 y 
~ OFFICE ; os 


OF PAT ge 
" \' RESEARCH AND EVALUATION 


“e« ‘ 
x ‘ 


~ 


N 
te) 
2) ‘ -: 
=) . , 
op ‘ 
< ; ¥ wy 
oa) e ‘ “» . 
& = ’ 


austin icepan gent shoe) district *® .= 


. t) 


\ anes aa 4 A ee ace il —— pret ony ener rueey | (oe) 
\ é x . 
aa, Ce 2 | ‘ ’ Pe ‘ 
; : F " _ 
4 s . ity 
. ~~ eat < 
- we . Fs 
s ws 7 a 
+ eo . ‘ a . é 
2 ’ ‘ 4 ces 
' : e - 
3 ¢. 2% “ 5 ‘ 
4 ae 
’ S 7 et 
¥ ¥, 
‘ 
’ 
te ~_ é 
oy ey r ¢ ? 
. * te ~ 
d & 4 + t ‘ # a4 4 
4 ‘ Vv . *° ‘ 
j = \ ' th 
7‘ | a rl , 
{ ; | i } * = 
. fs } . 
, i | 
-_* 
% | 7 . 
ies alee ™ j r mo 
. | as 
bad « 
e a 
. . ens 
> ’ ‘ / $ 
4 # ; ' 
i, 3 a { { ’ 
~ . e . 
ry 4 ‘ |, 
ie alee oem 4 \- 
* 
. * ‘3 - ¢ * t i é 
. . BOARD OF TRUSTEES 5 ths Al : 4 
4 k ‘ ’ : é ¥ . ‘ ts 
by i at os ' ‘ : : , ‘ ee 
: f Carole Keeton ‘McClellan, President P : ' 
? . * ¥ . F 2 oue if 
. 4 ; Gustavo L. Garcia, Vice President 
* ‘ . * ’ aol ' 
i” 4 ? " rid i 
: eS - Reverend Marvin C. Griffin, Secretary : 3 . 
; : : be -e 


aki ! a SRE PS, 2 

Will D. ska a ae DeCourcy Kelley a e, SRS 
‘ ; My 

; Jerry Nugent - ¢ M. K. Hage, Jr. * a, ra 


a . | . ae | | ss 
: a SUPERINTENDENT OF SCHOOLS 8s ge ee 
> ok z sg ' : / / : ’ r 
ow po ok i Dr. Jack L. Davidson ! ; 
‘ , * : ru . "6G ei 7 rs ‘ 
Say, ri _ — ‘ « ca ia . a : 
~* COORDINATOR, OFFICE OF RESEARCH AND EVALUATION 
~~ a ot é Dr. Freda M. Holley ; 
eco ie i : pbs ‘ < . r . ; “ 
> ‘ . a a ry 
of : * ‘s © ° 2 ‘ > 
* = (, P as ’ 1 te 
ewe OOO | Og ea Oe 
< ° 7, | a ' F 2 


' Evaluation in the Seventies: . 


_ What We Have Learned About 
x \: . = , ; , “ [aad (: 
ee Program Devetopment and Implementation aid 7 
iy ; \ 
+ 
a Py 


, Freda M. Holley 
4 


Austin Independent School District 


~ 

. 
“ 

t 

Pi 
> 
4 
. 
’ 
1 
\ 

¥ © 

AL ‘ 

« =  * 

a ¢ 


( ° j 
* i . 
p 
e\ . 
= ’ . : * 
¢ “i r . 
Me i he A 
Publication #75-57 . 
. 2. tes 


2 


J " = 4 \ i . 


° 


ca Evalieit'ion in the Seventies: 


What We Have Le rned About 


ay -_ Program Development’ a ee 
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: Ze . \ 5 
“ Need and’resource have met head-on in the. Seventies. Society | ; : re 
4 


at eat faces” the inexorable dddtaten’ “requ ved of a world with 
finite resources up against the wall of seemingly infinite human 
desires, and needs. New York fired policemen\ and Chicaga cut the :  # 


length of its adlivol year: Unthinkable acts ‘ few yd ago, but 


for@unners to.a mood in America ae eee daaets of the 
: \- 


electorate: suddenly hear and understand the ascetic messages of a 


yoyng Jerty Brown. \ \ \ 
\ 


This c¥ash between resources and needs in Texas and in Austin 
is muted, but nonetheless clear. Resources are starce and growing 


scarcer: utility bills soar transportation costs skyrocket, sal- 


* 


aries inflate, and taxpayers rebel. \ 


« 


7 


‘ But /the needs do not diminish. A government spokeman says: 


It can still be said, unequivocally, that the major: 
unresolved issue facing American’ education remains 
how to significantly/improve the education of. mi- 
nority and poor children. After years of social 
change, large scale funding, innovations of many 
types, and attempts at radical’‘departures in school 
governance, it is safe to conclude there are no 
panaceas, no quick and painless solutions, no sin- ~ 


gle strategy which can be’ counted upon to signifi- 


cantly enhance the educational experiences and 
opportunities of. minority children.” / 


"And the Austin Board of Trustees includes’ among its four top 


priorities for the year, the achievement of low socio-economic 


students. 


/ 


The country faces up to tHe. inescapable fact that ten 


years of declining: achievenerit test scores really do mean that 


students are less dennebent in Basic skilla: 


/ X 


2 


while Austin takes 


\ 


) 


its first long, surprised look at district teat scores below the 


national norms. 


‘Nationally and locally people discover that gifted 


students are indeed a neglected minority. In Boston and in Austin 


we are Pentel iy the court that minority isolation is meta: 


‘ALlegal, and by ers California that desegregation iy schools 


is no direct route to intereneien within * Seeidal: 3 


@ 


TRay C. Rist. 


‘ 


Race and schooling: Key policy issues. The 


a 


Educational Forum. Vol. 40, No. 4,May 1976, p. 514. 

2a. Harnishfeger agd-D. E. Wiley. Achievement Test Score Decline: 
Do we need to worry? St: Louis, Mo.: CEMREL, Inc., 1975. 

3Harold B. Gerard and Norman Miller. . How busing failed in River- 


side. Psychology todgy. Vol. 10, Tans 1976. f 100 


° 


And suddenly the meaning of educational accountability becomes» 
fully clear: We can no longer afford any but the best, the most 


effective people and programs’ and both must yield dollar-for-dollar 
valué, | 

x the public says reduce your budgets, and ideally educators 
“gnould be able to say, "Well, yes, if we must-, we will eliminate 


‘this! and keep 'that' because 'that' has higher aero for sts 
. 
dents. en 


{ 


| Unfortunately, today we’ cannot respond that’way with oak 


| 


) 
surety, because public stica ue has never really had to be a 


édabcotrectine industry, nor has it sought to be. Until very re- 


cently it could be said: ; 
|. failure to evaluate educational programs is a | 
/ shortcoming not limited to the Federal Government. ‘ 
. State and localities, which supply 95% of the 
pe funds for public education have done little to 

/ ' evaluate the effectiveness of their school apavene 


= and educational approaches.1 . | 
re ae! | 
Thas, one finds that research and evaluation units in public - | 
school are a most recent development. So recent in fact» that 
i 
© ‘ - t ‘ i 
TJahn, W. Evans, "Evaluating education programs - Are we getting 
a anywhere? piugey sone Researcher. Vol. 3, No. 8,’September 
* 19Th.. “P. 6, = - 
\ 1 
’ / 
i 
+ 
ci " 


a 


_ only now are effective technologies for defining or measuring 


either cost or effectiveness beginning to emerge.1° 
Moreover, educators, school board HEN DETA and communities 


do not understand the constraints on the Suruelaeiws of their 


, 


> new research and evaluation units, find such constraints unpleas- 


ant , and hence do not assist them in providing the eaninal con- 
ditions anaer qrigh such technology can derive cost-effectiveness 
answers.© puciaeah and avaicabtan units must thus inevitably 
struggle -not only with their own technological tpastenttonk. 

é \ 


Ithis is not the simple problem it would appear to/be on the sttr- 
face-since that "schooling industry" that currently constitutes 
1% of the entire American economy (the "education" industry 
which also includes such things as book production, libraries, 


* and museums: constitutes some 29%) actually has several production 


i) 


functions: knowledge, skill building, and certification in add¥- 
tion to two possibly even more important service functions, cus-. 
todial service (childsitting and keep young people off. the streets 
and out of the job market) and community center services (PTA, 
school plays, etc.). True productivity measures will have ulti- 
mately to encqmpass all these functions. Kenneth E. Boulaing, 
“The schooling industry as a possible pathological section of the 
American economy. Review of Educational Research. Vol. 42, No. 

1 Winter 1972. Pp. 129-190. : 


2T. R. Houston. The behavioral sciences impact effectiveness: 
model. In P.H. Rossi and W. William (Eds.), Evaluating social 
programs. New York: Seminar Press, 1972, P. 63. 


- Sinn M, Lee and Freda M. Holley. An ideal research design in | a 


public school setting: Or where are you Campbell and Stanley 

now that we need you? Paper presented at the Annual Meeting 

age the American Educational Research SORES SSIs Washington, 
» 1973. ’ 


or 


but often with ‘an alienated environment as well, hence postponing 


. 


the answers education needs. 
t ny 
Finally, in Austin as elsewhere, rather large sums oif money 


« 
have been expended to meet some of our most pressing educational . 
needs, particularly in recent years. Until'the Office of Research 

one ftyaliuation was begun three yoane ago, little or‘no ajatenviie 

taepops was “‘peihe nade to nienislbi the pont set feesivyenaas of these 


‘efforts. Since, this office began we have discovered little that 


» f 


has been ajfferentially effective. Such divpencntint effectiveness ww 
‘is the. first prenegivtte Since: $ extra/gain in effectiveness 

(over base program) = cost effectiveness of new popes When, 
for exanple, around $350,000 are spent ‘to add aides to.-classrooms 
and no. Avitticnt iaeeton in. knowledge gains are evidenced, no 


meaningful cost/effectiveness ;yratto can be calculated. Indeed, 0 


; 

gain in effectiveness = $350,000. This example to a large extent, 

however, descfibes the picture that we have found for numerous | 
A 

programs all around the ‘United States as:well as here’ in Austin.253 


A major reason for this appears to be our. lack of knowledge about 


lyJon F. Wergin. The evaluation of organizational policy making: 


A political mode. Reviewof Educational Research, Vol. 46, 
No. 1, Winter 1976, p. 82. 


2Allan C. Ornstéin and Barney Berlin. Social pare and federal, 


funding. Journal of Research and Development'in Education. 
Vol. Oy Noy. 3; 19755 Fp; 66-87. > # 


3As a side note at this point, it is well to.\note ‘ink Austin is 

not ‘alone in this situation. Austin has probably been far more 
effective in, its efforts than the majority of school districts 
around the country. One has only to visit ‘other ‘parts of the 
country or talk with educators from other ee to see 


aia educational improvement programs. The Sqventies have\ 
in fact given us our first’ 'parge scale research base om the organi- \, 


zational implementation of school district change programs. 


In fact our. greatest gabn to date from educational change 


efforts resulting from the 1965 federal ESEA legislation may be that 


out. of initial negative-findings can come an increased understanding — 


of the processes of educational improvement that will fead to more 


“effective intervention programs in future efforts: Perhaps, the 


greatest potential’ of evaluation must lie not so much in telling 


us where We have been as-in making our pursuit of the future a 


, 


‘more rational endeavor. — 


ey # 
. F aC i - 
' a n 


this. Moreover, parent, teacher and student surveys by ORE in 


. Austin gontinuously reveal all ‘these groups in‘the majority to be + 


positive and supportive in attitude. Research directors in other 
a@ities tell me this is not true in their surveys. However, Au&tin 
is at the forefront of cities that have chosen to honestly, crit- 
ically, and openly look at their efforts. In that process we lay 
open unfortunate truths that are endemic to all of education. F 
Having done this, however, Austin is in the position to make deci- 
sions on the basis of benefits to students rather than political, 
pressure-group-generated demands and thus to become a truly cost/ 


- effective ‘system. As a second aside it is aiso crucial to implore 


that’ federal and state governmental agencies not commit the mistake 
of withdrawing' resources from a‘district because of its willingness 
to gather and examine data about its activities. For the Office of 
Civil Rights to:withold @ll ESAA (desegregation aid) funds from a 
district because it has identified a disproportionate number of mi- 
nority students being disciplined and designed.an intervention 
program to address the obvious need is evidence m their part of 
‘decision-making which is neither rational nor of sound judgement. +- 
It is hardly an act to encourage accurate evaluation in sensitive 
areas by a school district. 


\ icles, C.H. Evaluation research. Englewood Cliffs, N.J. Prentice 


\ Hall, 1972. r 
\ 


. 


@ 


4 


\ The major intent of this paper! 16 to extract from the Office 


a Research, arnd Evaluation s ‘three years of acpartodee with eval- 


uating propran Axplenshtablen and from the rather extensive recent _. 


literature on this topic, an initial set of hypotheses about what 


° a 


a school district must do in “the impleméntation of@programs if the 
am we 
differential effectiveness factor in the cost/effectiveness’ ratio 
is to become other than nero. 

| These hypotheses are stated. belov in guideline form, but 


Bie the state of @ducation' s CUPEERY understanding of its fuactione 


and products , it. remains clear that they. can “be in truth no more 
> 


than working ierpothewtal’ However , our understanding ‘of our own?! 


“effectiveness is likely tg be enhanced by any theoretical approach — 


; to our activities regardless of how tentative it may be. 


This, it is postulated that in future Couepakaen efforts;. 


there will be an increased likelihood of ‘program success and effec- 


: pee 


tiveness to the extent’ AISD assures: 8 * 
l. ‘Priorities by which programs are approved or disapproved. 
2. An understanding by poth | top-level administrators and 
_ board members, as well as by program developers, staff 
members and evaluators of the organizational context 
and its characteristics by which programs live or die. 
3. An understanding, by those same groups of the program im- ° 
plementation process. 4 
- . 4, Realistic program designs: - 
Adequate evaluation and full use of evaluation findings. 


m 


@ 
* ‘ 


“a Li 


1This paper will actually be a first draft ,of a more elaborate 


statement to be prepared during the course of this year “under 
ORE's study of Low S-E-S Student Achievement. 


’ i : - 


.. 


me. 


that. only a very few new undertakings should be attempted at any one 


: . 2 , 
4 FJ 
* : : af x : . ; 


cy 


4 
The ances from whieh each of these Abaca have been 


derived and an amplification of their neeninks fallow. in: the sec- ’ 
eee below. — ; Fy Se eae, 
1. “PRIORITES 


a 


abe realities of implenention which make the installation of) 


even small scale programs a complex act in a school system dictates 


v 


time./ This principle, first documented from the literature in the 


- » . 


first year evaluations of ORE, seems no less valid in 1976 than in. “i , 
et ; 


1973--perhepageven more so. Although the limit ‘on new efdeavors 
proposed there was three, the list*of change eetorts impacting ® 
the school systen i “each ‘succeeding year has exceeded that number 


Sapatasealbier, The second corollary was that no less than three 


; years, should be allocated to the. full installation of a program. .. oe 


Thre. only major programs initiated in or after ‘1972-73 are 


considered new efforts, we have the following listing. (which could ° 
‘no doubt be added to without considerable effort ) of new programs 
_ operating in 1975-76: oe "Sh , 


Phe BSAA Programs (Basic, Pilat) Qo cate abe 


Community .School Programs 

State Compengatory Education Program to Pai a 

Sixth Grade Schools - cea aS nn 14 

Area Directors and, Area Organization Schema for "Elementary ~ 3! 
Education et 

Bilingual Education ‘Programs ( ESAA and Title vit) 

Plan A.Special Education Program 

New Administrative Evaluation System _ 0 

Quarter System a 

Evaluation/Accountability System — ees 


— 


at ' 
: . Py 


. ; \ ‘ : va) 
* 3 os 
e ry *. : “ ‘ 


‘New Central Office Organization (Plus considerable changes in : 
key administrative personnel) ace 7 : 
_ Basic Skills Reemphasis ae Z : 

: New Budgetfte-System, (Bulletin 679 and Zero Base Sudeatiun’: 2 
; ar , Managing Classroom” Positively oe Be iy. | ? 
oa - he effects. of this’ number of magéy innBvations sndwifent tena lEy 


"8 will -be described below ‘But, here it Should be noted’ that a 


number of new v progrannac efforts. are competing. for installation 
oe in. 1976-77. » Among those ‘Wnt pave been seShowed. are: tes 8 : 
a _ ~ A-New Teacher Evaluation ‘System a ee sae 2 ee 
r Gifted Program ae rr es 
-HighSchool fr the Peptovadteg Arts! 
i ; - Parenting Project (Title IVC) ; 


"A c) A M@Wority Recruitment Program 
evised Desegregation | Plan a a : 


ee ‘s Sane only three new programs" guideline is clearly an oversimplified 

one. The scope and potential ae of each new enterprise 4g a ya ° 
determinihg variable not taken into csnhtdeeavten by such a rule, 
In fact, HOWEVER, the BreReAme listed above are ais thaws a 
considerable district-wide or near district-wide Se Small 

. pilot i and federal da of lesser dimensions also exist; 


a . vw. oS *. 
| “but are nbt included. __ Re a | 

em . While the establisHing of priorities and the committment to ~ 

observing them when faced by the obvious enormity of needs from 


‘ 


_which such a list springs will not be an easy task, our failure to 


. e- Ni - 
do so ‘may only postpone and perhaps worsen the inevitable facing 


of the truth. The school system will not have the Bceouroess pars 


4 


2 : ' ticularly the ‘time PeBOURCER, to. ‘ieek all seedk through successful 


Aagtenenbabition of such efforts. . FE ase y ‘S 


; _ ¢ 


Hence, the setting of priorities and adherence to them in 
program adoption‘would seem to be 2 prerequisite for successful, . 
implementation. Where mandatory programs such as the aaa Sys- 
tem or desegregation exist, it is perhaps wiser to forget other \ 
change programs. — j 


ae - V 


2. ORGANIZATIONAL CONTEXT 


‘ 


' 


AISD. is an organization. Moreover it is a complex organiza— 


: ti ‘ 5 : Us 
tion consisting not only of an overall system but also of numbers 


“ar - 


of: linked subsystems it 


Iphis diagram first appeared in: F.M. Holley, J.£, Watkins, Jr., 
et.al. 1973-1974 Evaluation Report: Individually Guided Education 
(IGE) Program. Austin, Texas: Austin Independent School District, 
Office of Evaluation, 1974. i 


Ww 


af ne’ ‘4 


we. 


. School systems are to be defined as Sede pias because « 


They are rationally formulated, have officially stated 
goals, and contribute in: specific ways to the larger * 
system in yhich they are embedded. Schools, at least 
.. in part, socialize and allocate children so that even- 
* tually they will perform adult roles in the community 
adequately. 


Brograns or’ change efforts are aise ia in this orgenizationel 
context. The overall system must be’ receptive to the’ effort and 
the program will have to be incorporated into the organization .to 
as successful. A number of organizational Cc eer eres should 
sour be understood and taken into account in program design. These 
are covered in detail in sections A through E below. 
A. ‘GOALS mon c. Bat 

boal clarity Pela one to Se an‘ Amportant factor in organiza- 
tists Sera, A good analogy ree the public -sehool system sit- 


uation, however, mshi be found ina "group of, children sven two * 


. / 


- hours a jay ~ learn ' ‘ganes." - Several things ould result. First, 


4s 


somé children hay group ‘Sopebhen’ and learn, say, the total game 
© : 
of baseball. Another group ae eines to dearn the skills of pae= 


ing or kicking a football. still another single child may. elect 


tp toute dts personal interest and memorize the ruges of chess. 


It.-ia virtually certain, however, that we could have little assur-' 
os ; \ . i . e ; 
anee that the children would comé out with a common core of "game 


a 
4 ' 
f . 
’ 
“« 


. Joseph B. Giacquinta. Status risk-taking: A central Sent in the 


initiation and implementation of public school innovations. 
Journal of Research and Development in Education. Vol. 9, No. 1, 
ees P. 103. * ” re c ‘ 


< ‘ . 
‘ & x { 


* 


" 


learnings.’ Even more chaos might be expected if their parents Ps Ge F 


"and then "be a good ngs 


were in and out saying now, "learn to win, 
‘sport," their teachers, "learn ene eaten, and their friends, 
"let's just have fun." Absurd as this situation may seem, schools “he 
are in just ‘such a situation. torn between mandates to Heaeh mores _ : 
devetupnent, skill febuititet: knowledge gains, etc. with diverse 
_ Groups of any given day demanding very ssaaeia % emphases. 
“Por special program achooth this problem’ls magnified by 3! ideal 
competing programs--Title I saying bring up English reading scores a * 


while the bilingual program ts saying teach the child to oppreciate aS 


coat 
t e a ’ ' 7: 2 
bs chate: culture. oe hk: —  _ es eae t 


Such a multiplicity of goals decreases the staff's surety 
* about their organizational and iuntiettonad gcals. 
| A less humanistic analogy would be building an industrial . | : | 
ae and telling it to build cars, can pickles, and produce - ‘ | 
television film all with the same vorkers: and equipment! 
a sPBDERACRE IS s by ‘contrast adueatdcn has responded successfully 
E. ‘ = “to Mane mandates pores aeusurreae resources, The National Defense 
. . tee Act and ee — after Sputnik brought great, Science 
achievement gains; the falling away of this support has been fol- 
- lowed in the Seventies by steadily declining science sarotinents. 


and measured science knowledge.! : a s 


ne 


Ne Juaitn M. Sauls. Highlights and trends from national assessment: 

~ Changes in science achievement. 1969-1973. Paper presented at the 
‘ Annual Meeting of the American Educational Research Association, : 
San Francisco, 1970 ae . eine 


( 


The Austin ‘Board of a May 1, 1976, took™ay initial 


/ 
step. in the sche of Sneredline ss ale goal clarity by setting \z 


short tdrm’ priorities. ‘If actions .in holding to these peer dties 


= 


: and rejecting non-priority, activities follow and are atopreea by 

# top administrators, oars should Beata district personnel consid-_ 
erably. Proposed new activities set. out on~page | ‘9 for - 1976-17, 

couse be considered against” Lia priorities at this point. 

* anaes eerevte Big clarify goals:and priorities and te due these 

in program dieiediaede should, however, follow ‘in future years. 

B. PROGRAM STAFF NEEDS tat 84 : ee or 
Another view of apy organization would in tevuen dt as.a series 
of formally detined: roles and role paket tonalite: These -roles have > 
beak stabilized to a ecaupintie dager in school | systems. eke has 


the fortunate consequence that school doors doen every morn 


in tines ‘of considerable bepaizational stress. The congruent un-~ 


¥ ry 


“fetbanete consequence is. that change even ina positive direction © ' 


‘will be relatively difficult to achieve. - Thé introduction of any 
a 
type of innovation into, such’ a scene could be perceived as a pro- 5 


‘blem of veteaDS: risk taking"? since any changes will require ete 


: changes inte are to some degree. both potentially threatening and 


rewarding to staff members. ‘ 


~ 


e° : ' 


Igiacquinta, op. cit. Pp. 103-114., 


aie &, 


“Viewed in this ‘way organizational change Kecomes a drama where 


_human actors will behave in accordance with: the usual rules of » 


human behavior, There are then a number nt ic as to be ex- 
‘ 
pected whenever c ae is ctopsiecd : 
it is generall agreed that ‘hunané act first ‘out of self-in- — 
terest, How | this'self+interest is expressed, DOveNers is a fynction 
a degree of the state of needs of the dndividual. Maslow! telis 
us that the person's basic heeds such as those for phydfeas well- 


*~ being and safety come first, followed by emotional satisfaction and ' 


. acceptance, “Only when these are satisfied is the individual able to 


perform competently at higher ahi and congruently at. higher : 
intellectual levels. Se ie yer all 


a 41 


Yet in educational change efforts we find program staffs oper- 
-~ 
ating more frequently than not with almo&t no job security. It is 


: 4. | 
in fact the custom rather than the exception to find program staffs 
spending major fortions of their.time, and energy preparing a proposal 


which will assure their jobs for the coming year.* -Even 80, unfor- 


tunately, they customarily remain on Renberndoaks up until or past 
the time when. renewal of their jobs must come... The ESEA Title vit 


staff, for example, last year received approval of their program on 


nen! ener ey ; 7 ; ‘ 
lmaslow, A.H. A theory of motivation. In Behavioral Decisions in 
organizations, edited by Alver 0. Elbing. New York: SapEMs 

Foresman and Co., 1970. ,Pp. 366-382, 

Washington has recognized the problem but hardly came an with a 
.wise solution. They forbid the writing of proposals by program 
staff, leaving #he digtrict to decrease the effectiveness of on- 
going programs by using its own scarce resources (ESAA guidelines). 


—_— 


4 18" 
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/ 
July 16 for beginning date of July 1. Title I school staffs are 
traditionally-not certain of their jobs right up to the beginning 
"of wéhinal:, The) not sdinaetea result, of Sbikee, is that federally . 
funded postions Rave little role status. Not many competent persons sy 
will foe live with such conditions; they will seek greater security. 
The likelihood of successful programs iindey such staff conditions eck 


” 


_will nét be,high. "Sate 


wo 


For, the district Be a whole, hovever,, there will also be a de- - 


moralizing potential. “When a situation such as that “ardaing itn” 


' ESAA both tast year ane this year arises ,1 large number's of federal 
. programs. .staff members do find vhpanslyss without a ‘Job and the \ 
‘ school ee can make no vrovigion for Shen Both to then and to 
bas other members: of the regular staff who know them, “flowever , the’ 


implicit message is likely to be that ‘the status of the job is not 


a 


very high. ae 
On the other side of the coin, incompetent persons may be re- 
tained in or even transfered into federal. positions which are 8 


_fimes seen as less crucial. Fcux 
eh « x ° * 


vgr 


lpoth in 1976 and 1975, the Office of Civil Rights put the district 
on "hold" for the receipt of federal funds because of a question ' > 
of the district's compliance with civil rights regulations. In , 
1975 this hold resulted in the total loss of ESAA Bilingual Pro- 
gram funds and the delay in receipt of Basic ahd Pilot program 
funds. At this date (July 1976), the district remains this year * 
on a hold status. * 


* ‘ ' 


. 


. 
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‘ 


- An organization which ejther fails to rid itself of incompetence. 


or to sapada ecapetenes in mais ie lea change programs in trouble ( 
‘ 


, 


two ways: Internally from lack of competent staff and externally 


from lack of Genanienbtonad acceptance of the program. 
mA . 4 ae 
If indeed school coca ate as’ the literature suggests, change - 


MENT ALAA Sreanisnhtons the type of staff person then ate to be 


quecessful in innovation--resigting organizations can be deacribed 
‘4 ° ' 


. Sis 
as follows: at . ) 


It requires an unusual combination of qualities: acreative + 
programatic imagination; psychological security and a autono-. 

mous nature; an ability to trust others end to earn the trust 

of others; great energy and determination; a sense of timing; f 
‘skill in organizing; andia willingness and ability to be : 
machiavellian where that is what the situation requires.1 a i 

+ : : 
Finding and holding such staff seems unlikely if the status of pro- 


gram roles is 80 low, Ps 


C, PROGRAM STAFF AND DISTRICT STAFF INTERACTIONS 

The intérrelationships of regular and program staffs will be, 
in fact, of central importance. All change programs primarily Beak 
the change of regtiler staff by arcane staff in some way -- the 
regular staff is expected to improve SHEER ERC skills, incor- 
porate iam anions Swherbacds in their teaching, discipline students 


~ 
differently, etc. Such behavior changes Lameage occur at light 


iy eae bP 
lerbért A. Shephard. ee ae innovation -pro- 


ducing organizations. In The planning of change, édited by 
Warren G. Bennis, K.D.. Benne, and R.D. Chin. New York: Holt 


Rinehart, and Winston, Inc., 1969. Pe 52h. 


° 
- ws % 1 
5 \ . iJ 
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ne 
uh ; \ 
, cost whether ' Wont will occur at all will be a function both of the 


t 


. ‘souk of the change to the district staff and he way the district 
staff perceives the program staff. How they Dp ceive the start 
will be a function of the competence sit, ee a, ave seen, the 
‘role status of the program staff. \ 


The power ‘and aia relationships of the ne govern 


iy 4 staff relationships = a considerable degree. In AISD, program 
F 5 ~ 
e staffs are not in the authority line over school. staffs. The 


following description of a program in troubie has been observed? 


7 


in some AISD evaluations: ¢ 
i The assumption that ‘the...project could be implemented 
y effectively with minimal involvement ‘of the central office 
reflected a general lack of concern with authority and. 
jurisdiction. The project director's relationship to the 
principals and teachers in the schools were advisory in — 
nature. Inthe organizational structure of the school 
district ,his position was under the jurisdiction of the 
assistant superintendent for curriculum, who was highly 
critical of the...project. Given prevailing conditions, 
the project director was able ‘to monitor the project, 
offer suggestions, remind participants of project goals 
' and guidelines, and otherwise try to exert information 
leadership in project matters. He was ‘not, however, in 
a formal pretsion of authority.+ ‘ 


The net‘ results of this in such ar reenve has been that “local 
nd ra 


~~ 


as was often used to siemens aroire? goals. 


* 


e 4 


oy 


TRolana J. Pellegrin. Problems and assumptions in the imple- 
mentation of innovation. 
1 Education . Vol . 9 > No . 1 > 1975 . Pp . 98-99 . 


‘ 


‘ 


4 


’ 
® 


One potential solution to the seemingly inevitable conflict 
; ye 
that has so frequently arisen between school and program staff in 
¢ 
AISD would in fact be the type of program demonstrated in both the 


: r é 
Quarter System and the State Compensatory Education Program (SCE). 

ae 5 (es é : 
These programs literally have no "program staff" for their ‘primary 


components. The SCE prégram for @xample consists of the provision 


of materials and staff development. Both programs were derived under 


‘the direction of administrators with school authority. Attitude 


° 


data collected in the SCE program evaluation indicate great enthu-. 


_ Siasm for .and acceptance of the project .°»3 In the Quarter System, 


iines of ‘district authority clearly acted to effect’ the program and - 


. & » 
order.4 | 


implementgtion appears to be of a distinctly different and higher 


Lan ee og = 


ie oe ~ 


@joy Hester. State Compensatory Educ -76 Final: Evaluatio 
Reports Austin, Texas: Austin Tadependent ae District, Office 


”- Of Research and Evaluation, 1976. 


. 


Where is also, however, a necessity to nonitor such an effort in’ 
school based planning very carefully since we know so little about 
this process. There is a possibility that such a project can have 
these results: "Changes in behavior turned out to.be minor and not 
of the expected maghitude. What did happen was that structural 
changes were absorbed into customary: operational procedures and 
work routines. The pressures of teaching students and carryout on 
other usual school activities took priority over devising new ac- 
tivities in accordance with the new structural arrangements. Dis- 
cussions concerning the implementation of new structural elements 
which were tedious and time consuming did little but produce role 
overload, fatigue, and stress among staff members. The regular 
duties of teachers and administratogs made long and medium term 
planning difficult.' ".. Roland J. Pepe its Op. eit. FP. 9T. 


‘Mpaule, Matuszek. quarte r System 1975-76 Final Evaluation Report, 
Austin, Texas: Austin Independent School District, Office of Re- 


seach ‘and Evaluation, 1976. 


Bo ihe 


D. SCHOOL STAFF NEEDS 
| ‘ 


emphasize the needs of the district staff. As the "status risk- 


The above emphasis upon program staff needs is not. to de- 


\ 
\ 


z Ps : é : - 
: _, fj Baking" view of pee ial ghange emphasiz it will beat con= 


vad siderable personal risk that principals, teachers or other schooL 


+ staff undertake a change effort.+ Often, in fact, it is g 


change as necessary to avoid status ri 


staff rejecting an offer of "resotizces" 


‘atively. I have-been icid ty a principa : iat _a statement that 

| Parvenhpansen by his school staff ina bilingual progran vas , 
) : "optional" was clearly ridiculous. Community and district sanctions, 
although anottieiar, were nonetheless operating to negate such 4 
choice for the politically perceptive principal. However, for the os 
school staff the risk factor of faifing with.the innovation, par- é 


ticularly with the exposure to intensive evaluation, is also ex- 


tremely high. They will need to be continuously reassured about 


their status. Thus, school as well as program staff will require 


much Seay en bat Clee. atkenteGn and support. F 
‘ Once, again, among programs this year where strong staff . : 
support of efforts is seen would be the Quarter System and the 
TGiacquinta, ap. cit. 
a 7 
ee 
' a ee 


‘ 


+ 
* 


State Compensatory Education program. In both cases one suspects 
that since no program staft existed, rates responsibility and 
attention was assumed by magmas district administrative staff. 
One furtKer anapacks that new, Lovercwiantsie staff iin many programs 
may at least initially detract from program implementation. The 


authority attention appears in fact to have supportive effects, but. 


also gives needed goal clarity (again in @ system with a 

multiplicity of competing goals). 

. The attitudes of administrators in effect tell the, 
staff: how/sériously they should take project objec- 
tives. less the project seems to represent a 
district andsschool pridrity, teachers (or other 
staff) may not put in the extra effort and emotional 
investment necessary for successful implementation. 


E, ADMINISTRATIVE SUPPORT 

There eppearer to be clear tendency .in AISD as in other dis- 
tricts to underestimate these requirements of tonleve adminis 
trative support for a program's implementation. As is obvious 
from the above sections, such support must be far more aetive Shan 
simple verbal endorsenent. Board BaAtAEE, the superintendent and 
other ay level staff must actively devote. time to mee project in 
vartoum ways: \ athention to and moral support for Chusrenets of 


the program staff, intervention early-on in any developingtengions, 


. 


1paul Berman and M.W. MeLaughlin. > \ Implementation of educational » 


innovation. The Educational Forum. Vol. 40, No. 3, March. 1976. 
P.. 361... - 

ow 
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and cogtinuous. monitoring of the implementation schedule and process 


evaluation. An assessment of the staff strengths and particularly 


of the project director can be helpful in determining the extent 
#of such support requirement, but even for the strongest staff in 


any real change effort, substantial time commitments-of top adminis- 

_trators ‘appeat to be eSsential. At the school level,:congruently, 
r 4 : ; “ig 

principals have been’ identified as the "gate keepers of change;" 


implementation appears to. be totally dependent upon their Purpere: 
Here, perhaps more than anywhere else, is the major duces cue of 
- that prescription: no more chant three innovations at a time. Top" 


‘ 


‘level staff will have too’ little time do adaguanelysunet “the needs 


of many saaaunedond:. . ‘ 
3.. UNDERSTANDING THE INNOVATION AND INNOVATION PROCESS 

Tesetee the increasing literature on ene we know less 
about this than we must. . Several theories,are currently being ad- 
ten: however. — Gene — s Concerns Reged Adoption Model . is one., 
“ ; Te saa that i daca move through a series ue levels parallel 4 
“to ntencdel need levels vuessiet from an aaeihine focus on per- 
ee sonal coping concerns toward a Heads concern with ees 
and improvement of the iphueaiark ‘Each level has its own strategy 


and training requirements if the innovation is to move successfully 


through. each of the stages. 


. ak 


Stages of Concern and Typical Expressions 
of Concern About the Innovation * 


Stage of Concern Exprassions of Concern 


Unaware i don't know anything about It (the innovation) 


Awareness I have heard about the innovation, but | don't 
: know much about it. : ‘ 


. 
< ' isi . 
: 
1 


Exploration How much of my time would use df this innovation 
téke? | 


‘ . i Mn 
Early Trial _ | seem to be spénding all my “time in getting 
‘materipl ready for students, ‘. 
7 . e FH , ES w ; 
, 
Limited Impact 1 can now see how this innovatiof relates to 
other things | am doling. 
> A 
Maximum “Benet it I am:concetned about relating the effects of God: 
* this Innovation with what other instructors 
are -doing. aan , 


* : 
S 
« 


1) 
Renewal 


| am trying a variation in my use of the Inno- 


vation that looks like it fs going to result 
‘ In even greater effects. 


e- 


. 
Ss. ® 


* : 
stages: 


8 ‘ 2 
Another mofiél views the process as consisting of.three 
"initiation, implementation, and incorporation. 


Each of 
‘ 
@ these stages have different resource requirements. 


Typically, 
the last (stage is never reached; yet it is here finally that we 


would expect to find learner outcome. payoffs. This final stage in. 


ry 


ae? 
“This table has been taken from the following paper with pel gahoy 8 
permission: Gene Hall. The concerns-based adoption model: A 
-edevelopmental conceptualization of the adoption process within 
educational institutions. Paper presented at’ the American Educational | 
Research Association, Annual Meeting, (Chicago: 


Aprtl, 1974). : 


e 


a. 


" 


~ 


Yan & 


tal | > 
c ae Oe ‘ . , o.? 
\particular means the assumption of considerable resource burdens by 


the district, which are typically not planned for adequately, Where. 


projects are specifically planned ‘toward the contingency of the loss 
! ; % 4 


of funding as with the outside SCE program dt, the sixth grade schools 


there would seemto be an increased chanee_that long term pay-off 


mpsqush incorporation would occur’, 
Similarly. with the ey of the school resource situates 
(the melee portion of4pchool bideets go to fixed personnel ae 


t 


it seens unwi sd for funding agencies to stimulate and encourage 


’ programs on a sop “money basis where ‘Imcorporation in the ‘long. range 


is obviously not likely. ee of oe resource requirements | 
(Community Schools or Title vit), However, it seems equally unwise 
of. schoediey yavtnepreténbark, blindly, into. such projects. . 


‘A-related airrisuity is the: tendency of federal "and 

-state agencies to view thei contributions as seed 

money to be replaced by district funds if the program 

is a success. But school’ districts know that the 

typical cost of such programs ($100-$500.or more per | 
student per year) is beyond’ their ability to finance” *  . 
for the student body at.” large, and the use of district - 
funds for applying the intiovation to only:4 small . a 
“number of students raises: ‘serious ethical questions 

_ for a regulated public, utility. 


“oy a 
¢€ § a} { ; 


anette heoplindbenpetttiiet 


ae 
muy of each of the models. vee would “undoubtedly, yeler b benefit 


., tous in our errors at progra “evelopment , but ‘the truth remains 


7 


a * Ly Peer Fate 8 es — 1 avmpenyinnt ymin fe 


h 


iyohn Pincus. Incentives. for innovation in the Public sahooln, : 
‘Review of Educational Research. Vol. ant) No. 1, Winter 1974 
Pp.: Lig-115., ° Pe 127 e , a 


g 


that both din planning process nd the pereniel ‘ev ‘evaluation process 


¥ . 


ao eduicational improvement or change efforts is: an area of 


developing theory and concepts. While all Boe Literature quoted in 


c 4 | 


this ‘paper amply ‘iaacratealin ‘the attention “this | 8 heing given, 
® 4 


- 


nonetheless: ee he . 
' Innovation continues to be frustrated by the lack of 

technologies of planned change for use by prac itioners. 

At all: stages in planning and implementing innovation, 

there is a scarcity of theoretical principies and prac- : Re eM 
9 tices that would serve as guides to action. Each attempt ° , 

‘at innovation is, @n effect, a voyage on uncharted seas. Le ee 

t + : : : rar 


Another writer calls for conceptualizations, of the change process:.' ete Ad 


We need ‘aihgea pate, consultants, ‘and other experts: 
who know where and when to enter the ‘process and how 
to engage ‘school fa¢ulties in the types of problem 
identification and problem solving activities that 
will result in teachers devising effective and re- 
levant strategies fen meeting the needs of children,® 


The ‘understanding that we need in all educational improvement eae 


effort is that we do not Haye the answers . Our’ steps forward should’ 


“. 


i be as “tenative and painstaking as steps forward in a mtaberoveress 


quaking bdg. We need,to easarutiy document _ our steps in every ° 

effort and to pause frequently, as in this- paper, to add it all up j ; 
é and’ see what it says*about~future efforts.——Fai lure. shouldbe 8 

: . ts - : = ? , 7 


_Agonala J, Pellegrin. Op. cit. P. 100. , 


2pavid A. Shinan and Ann Lieberman. A es for schéol change. 
The Educational Forum. ‘Vol. 38, No. 4, May 1974. Pp. Wh1—bhs. 


7 


* 


_ prelude to success, not to inaction. What we , must be about is the 


evolving of an "inquiring system," an Sitendeabion in the RROCEsy 


- 


of self-renewal. 1 ' 


‘ 


9 


4, DESIGNING REALISTIC PROGRAMS ; 

5 At various points above it has been indicated that the scope 
and quality of a prouesii design greatly Gunes the likelihood of 
program success. By and large, AISD programs designs bend nesta “2 
exhibit the same types of prebpete Sdentatied in the Literature which 


are discussed below in auctions A - F, 


. 


i GRANDIOSE DESIGNS — | 

' a Needs vat enormous magnitude and complexity appear to. invite 

) programs of grandiose iudsetonn Anyone who has taught poor 
children fully, understands that a computer-based tutoring program 
for 15 minutes per day is onitkely to change. drastically the 
acttovenent patterns in lower socio-economic-status sien, What 


-we may not understand is that a program such as this ‘year's (as 


well indeed as next year's) Project Assist is unlikely to make a oi 


_ difference for the opposite reason. Into one program effort is | 


€ 


., - thrown a reading materials component, a math oe a drama 


sensei leertstasmnsopabentiarsssin Rossy at tanner eli omeitrtarssnp ie ihatet ecm an cage eee tap hyeantme ore 
program, a social work effort , teacher ataen., “teacher and aide staff i 


=e rea hegre epee eat Soe 


4 
- 


A 


ms cempatee nee tnirperdrnremteneromnneranmnangy 4 * . a 
a ‘ e u 


ne oe 


, ¢ ° J % . ° i 
lWilliamson, Hohn N.! The inquiring school: Toward a model of — 
organizational self-renewal. The Educational Forum. Vol. 38, 
; '-No. 4, May 1974. Pp. 393-410. 


ee * 
. Sopnstein and Berlin op. cit. P. 88. ; | . 
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+rnennerntnnnonnnT On e-recent-paper~ documenting the-failure—of -mational SOCK AD vou u ne rmnnenennnnininmnnvenn 


‘ 
development, efementary and secondary school levels. Unfortunately, 


‘es such multifaceted problem approaches rarely can be adequately in- 
plemented, as we ‘see demonstrated in this Year's Project Assist 


evaluation.+ ‘ ‘ ge eS 
For such an approach to succeed ... . it would have to be 
preceded and accompanied systematic planning enormously 


comprehensive in scope. thermore, the implementation of ~ 
radicaj revisions: in educational programs requires impressive 
financ support, assistance by many administrators and. 


specialists, acceptance by personnel, etc. ...Itis a’ 

curious fact that some educators believe that radical changes 

in education are easily accomplished even in short. periods of 

time although it is universally known that even minor changes 

in the schools are achieved with great difficulty .° 

: Thus , again of the intervention efforts attempted this year, the 
State_Compensatory Education Program with its provision of materials 

and limited Schocivapectfic staff development , both based on school- 
level planning and involvement, would appear ,to offer the greatest Ae 
potential in that it is a compromise between too little and too © lage : 
much, A i ee 

B, EXTENSIONS OF CURRENT INEFFECTIVE PRACTICES { _ 


' Despite their multivariate designs the grandiose programs. | = 


quite often exhibit few truly new or creative approaches. 


service programs one reason is’ declared to be that these programs 


1Nancy Jo Derby. ESAA III Pilot Project Assist 1975-76 Final 
Evaluation Report. Austin, Texas: Austin Independent School 


District, Office of Regearch and Evaluation, 1976. 
Pellegrin, op. cit, P. 96; me 
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j tempted to add gore Progrens of apeentialy the same type "2° 


exemplified the growth of REEPRURE RED: < Meeibiay and plant species 


grow by multiplication; ihe stout Gin Seowtan’ organizations ae 


& 


‘In AISD and eieedien the tendency is to design new programs 


much the same as: past ones. Rarely have programs been developed 


‘from a comprehensive search of the literature on potential programs; 


rather approaches are usually derived from extensions of existing. 


{ ' programs within the district or fiom the serendipitous knowledge of 


designers. Cost /benefit potentials have not Senurelbeal cal- 


‘culated as a part of the program design process. Where program 


design is charged to school staffs as in’ the State Compensatory 

Education diroesse the provision of such input information would 

appear to be a particularly anetul. step. The critical:necessity 

of such input 3 almost chillingly realized in the face of. yelisne= 

piteated Craters that Show the primary source of information re- 

lating to innovations, at. least at the elementary level, is ...the 

textbook salesperson.© 

Cc. SPECIFICITY OF PROGRAM GOALS AND PLANNING 
By and large innovative programs everywhere in past years have 


been stated with very general, abstract goals. Once again this is 


, . . . Li J . 


ern sie é ob —— Sirhan 


1pellegrin, op. cit. P. 96. 


2Donald C. Orlich, T.R. Ruff, and H.R. Hansen. Stalking Curri- 
culum: Or where do elementary principals learn nee new programs. 
Educational Leadership. Vol. 33, No. 8, May 1976. # BI cei 


a 


e 


a 


amply documented as a national problem in the literature: 
It was taken for granted that general goals could be ‘ , 
translated into specific objectives during the imple- 
mentation. phase. Behavioral changes at work, to. the = 
limited extent that they were foreseen, were assumed 
to take place. as required when operational problems 
arose, Similarly, it was thought that common under- 
standings of the nature, meanings, and objectives of — aa 7 
_ [the project] would develop and be maintained in each 
school after the implementation had begun.1 


awe 


These assumptions traditionally have not been accurate, and the 


results are that the goals are subject to interpretation by what is 


typically a new inexperienced staff. In AISD's own projects eval- 
aniad this year as in® past years, un even find rather major project 
staff changes even within the course of the year--Title VII, Titje 
“-1, and ESAA Project Assist. * Lacking specificity, each staff must 
rediscover--if possible--what the project should be doing. 


...problems did not “work themselves out" under operational 
conditions. Throughout the implementation year goals con- 
tinued to be stated in general terms. There was an inability ; 
' to relate the goals to what was being done by the staff. ae 
. Accordingly, familiar work rqutines were not fundamentally 
m changed. Shared views of /the project/ continued to be elusive. 


\ Indeed, the concept of [the project/ - itself continued to 
be vague and susceptible to a host of personal definitions., fs 
‘ ...New personnel added to the staff for the implementation 
year were poorly informed about project goals, even to the 
rnionnaprcennrnmninennnine POINt..that.some.hadnot evenread. the funding proposals OM ne ees ranmeisnmmmnnennneeeen 
other basic documents .@ : 


© - 


Even where relatively specific process and input objectives are 


initially stated, the next essential step is rarely taken. A quite 


' ” 


bs . 


--IRoland J. Pellegrin, op. cit. P. 96. 
. ° ®Ibid. Pp. 96-97. 
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specific day-by-day or week-by-week schedule on both-a long and short 


a range fade ehoutd be developed. Project staffs typically appear to 
_ go from crisis to crisis both from this lack of planning and from 
che’ deiaviolalap’ crabs discussed previously. Thus, even where 
* programs have well formulated process objectives, as with Title VII 
this year, few offthem are typically achieved. 1 
D. STAFF DEVELOPMENT A , | 
Pa J single program implemented in AISD depends to a greater 
or lesser degree upon staff development. Unfortunately, staff de- 
velopment, as seen both gn the three years of our evaluations and in theg 2: 


. 


national literature, suffers serious problems. _ One writer has called 


it “Education's disaster’ atea."* 


On the one hand, our district and 
others typically have no staff development model into which project 
staff development es mesh. This could be compared to teaching a 
subject without curriculum guides, selidee sequencing, a text series, - 
or any other cieiiecions Districtwide as a result we don't know 
where we've been, where we are going, or seemingly even sas we're” 


, doing with much surety. Program staff development activities over- 


lap, compete, and get lost. Needs assessment is fragmented and 


a HERE NE oN eee al ii area “ neces area fener PmGANADHieisacsonrineenslinteye URW aorririrea alannah AitrlnmntttmdervetetOvidtghiaaliirvesstiucieftenntithiesseniestemditttetienlrmieentiteiinednnideiidaiaeratin 
“ Ig1ynn D. Ligon. Final Evaluation Report: 1975-76 ESEA Title VII - 
Bilingual Project. Austin, Texas: dustin Independent School . 
“————~ “District, Office of Research and Evaluation, 1976. See erent 


2Lonnie Wagstaff and Tom McCollugh. Inservice education: Education's 
es disaster area. Admigistrators's Notebook. Vol. 21, No. 8, 1973. 
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mot often based on personal preferences rather than district ane . 
staff development is thus not systematically prioritized, and staff 
development is everywhere competing for resources of \eime. | 

* On the other hand, there truly appears to be nd reasonable. 
time when staff dudatiopuent can occur. Our evaluations have sys- 
tematically Snatneanbedl when it can't ocgur. Summers are out-- 
teachers go to sent or Europe, teach in summer school, or are sist: 
plain not interested. After school in the afternoon, everyone is 
too tired. Utilizing substitutes and pulling teachers from the | 
classroom hurts instruction. Teachers need to have parent con- 
ferences, plan, talk to other teachers, grade accumulated papers, sf 


i 


and do a score of other things on C-Days. 


Yet teachers and planners do say they need inservice (although . 
* planners and project ‘staff say it far more. vehemently and con= 
sistently than teachers and principals) while few of them ‘enhee on 
what it should be. 
| ‘A solution to these twin dilemmas of staff uajeiduuenh is far 
fee apparent, but it is iar’ that planning GrOneeas with major ‘of 
\ staff development components is futile unett some eaciubten of 


~~“ ghese-problem-does occurs 


E. STAFFING 
isthe The problems the staff will encounter and the superhuman, _ 
characteristics needed have-previously been laid out. . Program de- 


signs usually call for staff whose qualifications surpass even those, 


ce. 


however. TIGealistic designs call for a bilingual, curriculum de- 


velopment specialist of great skill, training, intellect, and in- 


‘terpersonal skills (Title VII), or-a project director skilled in 


' project management, proposal writing skill, aide training expertise, 


and specialization in’reading (Project Assist). Finding or re- 
Placing such staff--usually after the school year ‘begins because of 
late Hiditnigaceoula be little short of a miracle. Finally, the 
sdlary will typically be little above that for a classroom teacher * 


ae 4, alistic designs will be based on realistic staff- 
ing meanipiast “ ” 
F. OVERALL ‘ ; 

"ho sum up, the assessment of the realism of a projec’ design 
will utilize crtteria related to "the project's soenlextiy, cen- 
trality, consonance, ...the nature and amount of change required... 
and its implementation strategy." Every project design should 
undergo such assessment. A potential contribution that it appears 
the Office of Research and Evaluation might make to the district in 
the coming year would be to assist in aioe a model for staff 
and administrators to use in assessing or evaluating project designs 


| , 
using Buch criterias” 


5. EVALUATI ON 


One area in -which-ORE feels it has learned a great. deal is in 


wea 


how evaluation can best be conducted and used in program improvement. 


* 


“ 


, es 
Be 
wt 
~ 
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The evaluation process, we feel must: . I 


.be accountability focused. ~ s ; 
- . sbe rigorous fn design. a 
-collect implementation data as well as outcome data. 


In its use phase, we feel it should: 


-be a part of a systemwide, common planning approach. 
.be focused an real decisions ultimately to be made. 
-lead to recommendations and actions. ‘ 
.be carried out for the organization’ s top priority concerns. 
b 
On the negative side, there ard two things that we should not abc 


.Contract evaluations to qutside firms.. - 


-Conduct evaluation where; resources are inadequate. 


A. ACCOUNTABILITY FOCUS 


The Office of Research and Evaluation began evaluating programs — 
‘e 


* 


"in 1973-74 with a dual focus 3ervice to programs through ongoing 


"formative" evaluation and accountability information on programs 
to AISD and its public through "summative" evaluationr - 

To promote the service function, close and constant interaction 
with program staff was envisioned. It was Ebnunkt that the program 
staff should participate in evaluation staff selection, and that 
mratoatton staff should participate in program planning sessions. 
There vas ove some thought 4f housing progr and evaluation staff 
together ;. in one program (Project Assist) proj ct and evaluation 
staff were in fact housed in the same room. 


“After three years of trial and effort, ORE has come to believe 


£ ‘ 


that functions of evaluation are best accomplished nian 


andé procedures which are "accountability focused." The reasons for 
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this are complex and have been explicated elsewhere.2 
It may be said, however, that even the "formative function" 
appears to be served most usefully in the sense ‘that a program and 
staff which fully understands that the evaluation unit is peeeenbe 
‘ * ww» gbility ie will be more aidiniy veteneat of the program im- 


lementation\ ; 


(The accountability function demands an evaluation unit which 
is somewhat detached from the program and school staff. The danger 
‘in the integral involvement vith program staff is twofold. a 
& First, the ees must always guard against being co-opted = . 
‘by the program, thus losing his own objectivity. The danger here, 
of re lies in too close identification with ha Beas: 
Sidney Marland criticized early federal program implementation be- 
‘° > cause either programs were not etaluated or because the program 
- directors thenselves were permitted to carry out the evaluations. 
* In speaking on the efforts of the program directors he recognized 


‘e 
their human tendancy: "T must assume they possessed a natural and 


healthy bias in favor of their own enterprises." Evaluators are 


. 


no less human.’ Moreover, the possibility of direct intervention of 


cnvivecnwnnnime the program staff in the censorhip or concéalment of data is less... 


wa 


‘ likely to occur under the independent arrangement. 


s 
s . 


lfreda M. Holley, et. al. Models for the delivery of school district z 
evaluation: Service or accountability? Paper presented to the Annual 
Meeting of the American Education Research Association, San Francisco, 
1976. . ® 
. a F 
2Sidney P. ie a A responsible stewardship. Phi Delta 
1972, 54. P. 88. at i by 
at ; ad . . . * 
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Second, the program staff may suffer a very real sense of be- 

: trayal and alienation when one? are faced with an dice 
report" from an evaluator with ~wh6n they have interacted as @ co- 

‘ worker and Wgervice" evaluator. Indicators are that staff members 
can more clearly accept and interpret papdoaniate: however negative, .; 
from a source whose function is unitarily defined than from one 

regarded as a dichotomous source. The reasons for this are best 


explicated + a quotation from Maslow: gs 
"In may early years in teaching, I certainly loved 

; my students and felt very close to them and wanted . i 

* to be buddies with them. I learned only slowly 
that while I could keep my smiles and friendliness : iS 
and so on separated from the grades, i.e., I could ae ; - 
certainly love somebody who wasn't a very good 
student of psychology, they rarely could accept and 
understand this. Normally, when I was friends with 
students, they felt I had betrayed then if they got 
bad grades. They thought of me eas a hypocrite, as a 
turncoat. .. . .Slowly I had to give it up; until 
‘i now, especially in large classes, I keep my distance 

and maintain English-style relationhips rather than 
getting very close and buddylike."2 


*, 
A final important factor with respect to the choice of the account- 
. ability focus is an admission of the lack of facility with which an a. 


evaluation unit can provide "formative" information. In reality : 
\ : the process of collecting and analyzing data is so time consuming 


riewntrnmmunconsonts hat--feedbac k-can-rarely.be_given.to_a_program staff in sufficient ge iar a 


~~ F 


-) 
a “4 ee * . ‘ 


. . 


labraham Maslow, as quoted in Gene V. Glass. A paradox about ex- 
» cellence of schools and the people in them. Educational Researcher, 
Vol. 4, No. 3s March 1975. P. 10 


- e ‘ 


\ ; ' 


time to guide their efforts with much: facility. Moreover, if a staff 


feels. relieved of the monitoring function, they may be less alert 

and intuitive regarding implenentation processes. hile an aa 
akon deaten absolutely myet eall for, enGugh process data to ver- a) © 
ity the existence and qpeity of: the inlementation, the major re- 


quirement for ongoing program aandtovtne appears to pe more effi- 


< . 


ciently left with the staff. Of course; dacs from the 
evaluation should still be provided to. staff as it -becomes avail- 


t . , . @ 


able... ' * . 
B. RIGOROUS DESIGN | ww a 
All oF literature and our ow experience aipeeate that the 
more rigorous thie evaluation design, the more parted we_can be of 
our answers. The prime definito# of rigor is quite simply "random 
assignment" to treatment and alternative or control treatments. 5 8 
We are consistently told by the literature that this is SHRERESED } 
we are consistently unable to -carry this out in ‘the educational: 
setting. The reasons are of two BSED First, the autonomy of 
schools to aeinee "treatments" is heenbened by randomization. 
Second, it is held that we cannot deny students "good" rsavnnnt, 
Both of these arguments are questionable. The tiwle anmueptions 
‘are that the treataents: is both beneficial and available to all vho 


need it.1 Both of these assumptions are in’ fact EaAee With 


Siesta thie EASA NAPA A RAR 
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help students, in view of this data (admittedly inconclusive at © “ait 


oma 2a 
a 7 


q 
oS 
te 
\ 4, 


that it will produce better results th n.antther. There is, how- 
ever, evidence that any new treatment -ot any new educational ac- 


tivity has a chance of being harmful: 


Thé purpose of evaluation is to ascettain whether or 
not the treatment is beneficial, There are numerous 
cases in which supposedlv helpful treatments have 

been found to be harmful. 


to mount up in our epateattann that ' ‘the more\adults- renant in a 


WG 


classroom the ‘lower the pupil achievemagt." Although it has aed 


declared unthinkable to "deny classroom aides" to help teachers . 


this time) the’ possibility clearly exists that we may be hurting 


patients without researching its effects. 


student achievement by adding those aides. The parallel to this. 


situation is how we might react if this were a drug being used on all 


With respect te the selection factor: . a,” 2 > ge 


Given that all those in, need cannot possibly be given 
‘the treatment due to its scarcity, random selection 
of respondents for treatments is not only scientif- 
ically but also ethnically unbiased and thus is as. Be os 
fair a-procedure as ‘possible. Or, given that the *® 
treatment cannot be administered simultaneously to 

\ ali who need it, some respondents can be randomly: 
selected to receive it a ie with those who receive 
late G as ia 


9 


Pred tieaniataimrtapneceiriperiotenMerenihieceenecatintetbneesibvign 


‘ 


ei 


om 


saulgrinnt, aces certain we can be sont che: ankwets we de- 
7 a 2 


rive, This does not mean our. answer will be untrue-or evensun= 0°! 


iat J 


| 
. + . ry a Son”. 4 F 
\. - . t ty ae of i : : 
. * ‘ . 


ry = . oo . a. 


lpaul M. Wortman. Evaluation research. American Psychologist. iar 


‘Vol. 30, No. 5, May 1975. P. 570. 
2Ibid. P. 571. + 
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- * , reliable, ‘but certainty about these will be reduced. Evaluation 
*» j 
has some techniques (time series designs, comparison groups, post’. 


hoc datchines etc. ) that can ie used, but there is no good sub- | 


: a ae 
BB kg datitute from random.assignment. Evaluation nae he nadie wand “whitch * 


rogat . 


can salvage answers from bad program duvteon: 


. &. ae ARATII A i | 
ey gs , : . This entire paper {s in ‘tact, documentation, for the necessity 
a age implementation data collection. Unless we know what our treat~. 
ment or program really sanclees ait our outcome data will sane 


bi cai +€ ey : ° 3 : ’ . : 
D.: $YSTEMWIDE PLANNING =| 


pe vat foe eS es 
A great deat of euckey and effort is expended on planning in 


: ‘aISD. The sab teats ‘aia budgets are formulated peelpa ‘set 
objectives, aeapeeale get written. The problem with that planning ' 


is, first, no formal model integrating ene planning process exists: and — 


second, that it is not centrally coordinated or  fionttored, Mobeni 


. ; there ‘is no systematic détinition of. the feedback data that is to be 
: , : “used in the vaptntis planning, ‘efforts. ; . . " £ 
: Until data feedback judgments are iicadits incorporated as a 
~ part: of the planning briceeg evaluation data will not. teach its 
| acon potential. ‘ | ee 
Os bat Be PRIORITIES. 
. “a "| The most feportand dectetons fackon the system require the - 


1 


“best dhta as input. Moreover, since evaluation is an expensive 


; , ad | i # : - Z Y , | a AL se, . : at 


o si Zz 

- ‘e 1 
process, it should be directed at ane most’ important decisions. 
From both directions, then, comes’ the requirement that priorities 


should be set for both decision diadbtione: and evaluation foci. 


FF. RECOMMENDATIONS AND ACTIONS : 


‘ : * * 


In 1975-76, ORE made recommendations to deci son-makers 


regarding the information collected. A far more desirayge route 


ds ‘being followed this year with the recommendations to come from 
. ‘the school staff. How well this works will have to be assersed in 


terms of the eventual actions taken. 


. S z ‘ 7, 


“G. CONTRACTED EVALUATION 


_ORE has interfaced with outside evaluation contractors each 
year-‘since we began. The interface with the contractor almost ie 
ways: cost as much in district time and resources as. aia the eval- 
uation contract in money. Rarely have the contracts resulted in 
high-quality “evaluations. Moreover, the lack of know of the + 
contractor of ta systen and program context ay lead to a 
pretations of data. When the contractor leaves, he takes with him 


the reservoir of program knowlege he has gained and it cannot 


benefit future program designs or evaluation efforts. 


Contracted audits of AIS5D evaluation efforts would however { _be 


desirable in much the same pay-off framework that leads us to, have 
financial audits. 
H. EVALUATIONS WITH INADEQUATE RESOURCES . 2% 


: There is a minimum resource iis for an evaluation below # 


which the. quality and comprehensiveness of the eveIHENa On is so Iow 


as to be dangerous. Theré comes a point when we should eran dl 
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menuas to carry out ‘an evaluation because of ek factokas. That 


Oi setae was Suaenad vith the evaluation of the ESAA programs this 


coming year. Although the Bren for evaluation is high, the infor- ” & 
mation that could have hema collected with the ‘proposed resources 
would have been ihadequate bes result in better Aéptetcis,. The sane 


is true of the proposed resources to evaluate the Community School 
J \ a a 

effort. Increasingly, ORE will be trying to increase the cost/ef- 

fectiveness ofCevaluation activities while yet paying as much heed 


to thé’ potential effectiveness factor as to the cost. 


CONCLUSION 4 


ib 8 5 ; 
In summary, from the literature and from ORE evaluations comes 
substantial support for the hypotheses that the route ‘to improved 


—programs in AISD will be enhanced by: 
1. Setting priorities by which programs are er or . 
'  . disapproved. 
2. An understanding of the district as an organization. 
3. An understanding of the program seh a Sak process. 
4, Realistic program designs. 
5. Maximum use of evaluation information. 
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